Ubiquitous natural user system for human-machine interaction

ABSTRACT

A system is provided that includes sensor(s) configured to provide sensed input including measurements of motion of a user during performance of a task, and in which the motion may include a gesture performed in one of a plurality of 3D zones in an environment of the user that are defined to accept respective, distinct gestures. A front-end system may receive and process the sensed input including the measurements to identify the gesture and from the gesture, identify operations of an electronic resource. The front-end system may identify the gesture based on the one of the plurality of 3D zones in which the gesture is performed. The front-end system may then form and communicate an input to cause the electronic resource to perform the operations and produce an output. And the front-end system may receive the output from the electronic resource, and communicate the output to a display device.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 13/903,242, entitled: Ubiquitous Natural UserSystem, and is also related to U.S. patent application Ser. No.13/903,252, entitled: Tracking a User to Support Tasks Performed onComplex-System Components, both of which filed May 28, 2013. Thecontents of both of the aforementioned are incorporated by reference intheir entireties.

TECHNOLOGICAL FIELD

The present disclosure relates generally to human-machine interactionand, in particular, to human-machine interaction through an interfaceintegrated within a user's physical, real-world environment to supporttask-based activity.

BACKGROUND

Interfaces for human-machine interaction have evolved substantially overthe past decades. Even within the narrower field of control ofcomputers, interfaces have evolved from command lines to graphical userinterfaces requiring the use of a mouse or similar type of pointingdevice for selecting graphical icons displayed to a user. More recently,touchscreen devices have gained popularity as they open up thepossibility of gesture-based control.

Virtual reality is another approach to human-machine interaction inwhich an “immersive” virtual (or unreal) environment is created toprovide a user with the sense of being totally immersed in anartificial, three-dimensional world that is generated by computersoftware. A related, more recent approach to human-machine interactionis augmented reality in which a view of a user's physical, real-worldenvironment may be augmented with virtual elements. Virtual reality andaugmented reality are often implemented through the use of displayhardware such as head-mounted systems, computer screens and the like.But past approaches to implement virtual and augmented reality throughsuch display devices have met with varied, but suboptimal success.

Therefore, it may be desirable to have a system and method that improveson existing approaches to human-machine interaction.

BRIEF SUMMARY

As indicated in the background section above, past approaches toimplement virtual and augmented reality systems have met with varied,but suboptimal success. For example, significant difficulties remain inassociating sensor inputs to relative contextual information from a useror their physical, real-world environment. Further, current technologieslack a means to leverage associations between the user and theirenvironment, which in a number of contexts may implicate a particularrole of the user.

Example implementations of the present invention are generally directedto an improved system and method for human-machine interaction with oneor more electronic resources through an interface integrated within auser's physical, real-world environment. The system of exampleimplementations, referred to at times herein as the ubiquitous naturaluser (UNU) system, integrates sensor systems, mobile computing andnetworked systems to provide a new user interface that leverages anintuitive and natural interaction model and interactive modalities. TheUNU system of example implementations may form part of an augmentedreality system that includes a personal display system such as augmentedreality (AR) glasses including a display device, which may enablehands-free interaction and collaborative information sharing. In someexamples, the UNU system may employ other active or passive personaldisplay systems.

In accordance with example implementations, a worker may perform a taskon a complex-system part in their environment. The system and method ofexample implementations integrate sensor systems, mobile computing andnetworked systems to sense the worker, their environment and/or theirinteraction with their environment. The system and method may theninteract with one or more electronic resources based on the user,environment or interaction to support the same or another workerperforming the same or another task.

According to one aspect of example implementations, the UNU systemincludes one or more sensors and a front-end system coupled to thesensor(s), and perhaps also a display device (although one may insteadbe merely in communication with the front-end system). The sensor(s) areconfigured to provide sensed input including measurements of motion of auser during performance of a task by the user. In accordance with thisaspect, the motion may include a gesture performed in athree-dimensional (3D) zone in an environment of the user, and in whichthe 3D zone may be one of a plurality of 3D zones in the environmentthat are defined to accept respective, distinct gestures of the user.

The front-end system may be configured to receive and process the sensedinput including the measurements to identify the gesture and from thegesture, identify operations of an electronic resource. In this regard,the front-end system may be configured to identify the gesture based onthe one of the plurality of 3D zones in which the gesture is performed.The front-end system may then be configured to form and communicate aninput to cause the electronic resource to perform the operations andproduce an output. And the front-end system may be configured to receivethe output from the electronic resource, and communicate the output to adisplay device, audio output device or haptic sensor.

In some examples, the front-end system being configured to receive andprocess the sensed input may include being configured to receive andprocess the sensed input including the measurements to identify thegesture further based on the task. In these examples, one or more of theplurality of 3D zones or one or more of respective, distinct gesturesthat the plurality of 3D zones are defined to accept may be differentfor different tasks.

In some examples, the front-end system being configured to receive andprocess the sensed input may include being configured to receive andprocess the sensed input including the measurements to identify thegesture further based on the user. In these examples, one or more of theplurality of 3D zones or one or more of respective, distinct gesturesthat the plurality of 3D zones are defined to accept may be differentfor different users. And in some further examples, one or more of theplurality of 3D zones may be different for different users, with each ofthe respective 3D zone(s) having a different position, shape or size (inthe environment) for different users.

In some examples, the gesture may be a first gesture, and the motionfurther may include a second gesture performed in the environment of theuser and that is distinct from the respective, distinct gestures thatthe plurality of 3D zones are defined to accept. In these examples, thefront-end system being configured to receive and process the sensedinput may include being configured to receive and process the sensedinput to further identify the second gesture. Here, the operations ofthe electronic resource may include operations identified from the firstgesture and operations identified from the second gesture, with thefront-end system being configured to identify the second gesture withoutregard to the plurality of 3D zones.

In some examples, the front-end system being configured to communicatethe output may include being configured to communicate the output to thedisplay device. In these examples, the output may be communicated in oneof a plurality of distinct desktop environments displayable inrespective facets of a three-dimensional, multifaceted graphical userinterface (GUI). In these or other similar examples, the output may becommunicated to effect rotation of the three-dimensional, multifacetedGUI, or to effect selection of one of the plurality of distinct desktopenvironments.

In some examples, the sensor(s) being configured to provide sensed inputmay include being configured to provide sensed input further includingother measurements of the motion and/or orientation of the user duringperformance of the task by the user to work a complex-system component.In these examples, the front-end system being configured to receive andprocess the sensed input may include being configured to receive andprocess the sensed input further including the other measurements toidentify a known pattern that indicates a significance of the sensedinput from which to identify other operations of the electronicresource. Also in these examples, the front-end system being configuredto form and communicate the input may include being configured to formand communicate the input to cause the electronic resource to furtherperform the other operations and produce another output, with the otheroperations including determination of an action of the user orcalculation of a process variable related to performance of the task,from the measurements. And in these examples, the front-end system beingconfigured to receive the output may include being configured to furtherreceive the other output from the electronic resource, and communicatethe other output to the display device, audio output device or hapticsensor.

In other aspects of example implementations, an improved method andcomputer-readable storage medium are provided for human-machineinteraction. The features, functions and advantages discussed herein maybe achieved independently in various example implementations or may becombined in yet other example implementations further details of whichmay be seen with reference to the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWING(S)

Having thus described example implementations of the disclosure ingeneral terms, reference will now be made to the accompanying drawings,which are not necessarily drawn to scale, and wherein:

FIG. 1 is an illustration of an ubiquitous natural user (UNU) system inaccordance with an example implementation;

FIG. 2 is an illustration of one example implementation of the UNUsystem in the form of an augmented reality system;

FIG. 3 is an illustration of a front-end system in accordance with anexample implementation;

FIGS. 4, 5, 6 and 7 illustrate a sensor system, analysis system,execution system and evaluation system, respectively, in accordance withan example implementation;

FIGS. 8, 9 and 10 illustrate an example of a three-dimensional (3D),multifaceted graphical user interface (GUI), and gestures by which auser may interact with the GUI;

FIG. 11 is an illustration of a flow diagram of aircraft production andservice methodology according to one example implementation; and

FIG. 12 is an illustration of a block diagram of an aircraft accordingto one example implementation.

DETAILED DESCRIPTION

Some implementations of the present disclosure will now be describedmore fully hereinafter with reference to the accompanying drawings, inwhich some, but not all implementations of the disclosure are shown.Indeed, various implementations of the disclosure may be embodied inmany different forms and should not be construed as limited to theimplementations set forth herein; rather, these example implementationsare provided so that this disclosure will be thorough and complete, andwill fully convey the scope of the disclosure to those skilled in theart. For example, unless otherwise indicated, reference something asbeing a first, second or the like should not be construed to imply aparticular order. Also, something may be described as being abovesomething else (unless otherwise indicated) may instead be below, andvice versa; and similarly, something described as being to the left ofsomething else may instead be to the right, and vice versa. Likereference numerals refer to like elements throughout.

Example implementations of the present invention relate generally to aubiquitous natural user (UNU) system for interacting with one or moreelectronic resources. Some example implementations may relate generallyto a system and method for tracking a user (e.g., worker) to supporttask-based activity such as that including tasks performed on componentsof a complex system, and may be further integrated with a UNU system.And some example implementations may relate generally to The UNU systemmay be generally configured to sense a user, the user's environmentand/or the user's interaction with their environment. Informationcharacteristic of the user, environment or interaction may be analyzedto identify one or more operations that one or more electronic resourcesmay be caused to perform to thereby interact with the respectiveresources. The operations(s) may be executed and may be fed back toprior analysis for further evaluation in order to improve future resultsand provide timely information for any required or otherwise desiredoperational changes.

FIG. 1 illustrates a UNU system 100 according to example implementationsof the present disclosure. The UNU system may include, for example, afront-end system 102 connected or otherwise coupled to one or moreinterfaces for transmitting, receiving and/or outputting information.The interfaces may include one or more communications interfaces 104,sensors 106 and/or user interfaces. The communications interface may beconfigured to transmit and/or receive information, such as to and/orfrom other elements of the UNU system, one or more resource host systems108 and the like.

The communications interface 104 may be configured to transmit and/orreceive information by physical (wired), and/or wireless communicationslinks. These wireless communication links in particular may beconfigured to implement any of a number of different radio accesstechnologies such as any of a number of 3GPP or 4GPP radio accesstechnologies, UMTS UTRA, GSM radio access technologies, CDMA 2000 radioaccess technologies, WLANs (e.g., IEEE 802.xx, e.g., 802.11a, 802.11b,802.11g, 802.11n), WiMAX, IEEE 802.16, wireless PANs (WPANs) (e.g., IEEE802.15, Bluetooth®, low power versions of Bluetooth®, IrDA, UWB, Wibree,Zigbee®), near-field communication technologies, and the like.

The front-end system 102 may also be coupled to one or more sensors 106configured to provide sensed input to the UNU system 100. The sensors,and in some examples the communications interface 104, may generallyprovide input necessary for the front-end system to detect, recognizeand drive the display of a natural user interface for interaction withvarious electronic resources. Natural user interfaces may removeextraneous devices and input modalities, and may allow geo-temporallyand task salient interactions tailored to the role (or clearance level)and purview of the user.

The sensors 106 may include onboard sensors, off-board sensors, sensorconcentrators, sensor aggregators and the like. Examples of suitabletypes of sensors include user physiological-condition sensors, useraction-tracking sensors, user cognition-tracking sensors,environmental-condition sensors, environmental-tracking sensors and thelike.

Examples of suitable user physiological-condition sensors includetemperature sensors (e.g., thermometers, thermocouples), heart ratesensors, respiration sensors, perspiration sensors, muscle-activity(e.g., EMG—electromyography, MMG—mechanomyography) sensors, strainsensors, blood pressure sensors and the like. Other examples of suitablephysiological-condition sensors include image sensors such as cameras(e.g., digital camera, infrared camera, thermal camera, depth-aware orrange camera, stereo camera), audio sensors (e.g., microphones) and thelike. Examples of suitable user action-tracking sensors include positionsensors (e.g., GPS, inertial navigation, Wi-Fi-based positioning,RFID-based positioning), motion sensors (e.g., accelerometers),orientation sensors (e.g., gyroscopes, compasses, magnetometers),inertial measurement units (IMUs), pressure sensors, muscle-activitysensors and the like. Other examples of suitable user action-trackingsensors include image sensors (e.g., cameras), audio sensors and thelike. Examples of suitable user cognition-tracking sensors includeattention sensors, focus sensors and the like, at least some of whichmay include one or more sensors for a set of user actions that may beprocessed according to appropriate algorithms (e.g., micro expressions,gestures, comparative motion relative to task).

Examples of suitable environmental-condition sensors include imagesensors (e.g., cameras), audio sensors, position sensors, clocks,calendars, temperature sensors, humidity sensors, light sensors,distance/depth sensors, three-dimensional (3D) scanners, lasermeasurement sensors and the like. Examples of suitableenvironmental-tracking sensors include image sensors, audio sensors,calendars, clocks, light sensors and the like. Notably, an example ofone type of sensor 106 may be equally an example of another type ofsensor as various sensors may be used for different purposes. Otherexamples of suitable sensors include tilt sensors, landmark systems,dynamometers, telemetry sensors, fiber optic sensors and the like. Evenfurther, in some examples, the communications interface 104 may functionas a sensor for providing information from one or more electronicresources such as a local cache or backend information system.

The front-end system 102 may also be coupled to one or more userinterfaces for outputting and/or inputting information, such as adisplay device 110, audio output device(s) such as speaker(s),headphones and the like, haptic sensor(s) configured to provide tactilefeedback such as forces, vibrations or motions. The display device maybe configured to present or otherwise display information to a user, andin some examples may include the display device of a wearable (e.g.,head-mounted) or handheld personal display system. Examples of suitablepersonal display systems may include private, private-shared (linkedprivate) or public personal display systems such as those provided inthe form of eyeglasses, safety goggles, contact lenses and the like,image projectors, video projectors, any of a number of other active orpassive display systems, laser pointers and the like. In other examples,the display device may include a more conventional display device suchas a liquid crystal display (LCD), light-emitting diode display (LED),plasma display panel (PDP) and the like, which may or may not take theform of a personal display system (e.g., smartphone, tablet computer).

The user interfaces may also include one or more user input interfaces.These user input interfaces may be wired or wireless, and may beconfigured to receive information from a user into the UNU system 100,such as for processing, storage and/or display. Suitable examples ofuser input interfaces include a microphone, image or video capturedevice, keyboard or keypad, joystick, touch-sensitive surface (separatefrom or integrated into a touchscreen), biometric sensor and the like.Similar to the communications interface, in some examples, a user inputinterface may function as a sensor for providing information directlyfrom a user.

As suggested above, the UNU system 100 of example implementations may beconfigured to communicate with one or more resource host systems 108,which may be configured to host electronic resources. Some examples ofhost systems may be remote from the UNU system such as across one ormore networks, while other examples of host systems may be local to theUNU system. The electronic resources may vary depending on applicationof the UNU system, although examples of suitable electronic resourcesmay generally include software-based systems. In some examples, theresource host system may include an appropriate storage accessible by anelectronic resource, such as file storage, database storage, cloudstorage and the like.

Examples of suitable software-based systems (electronic resources)include information systems, building management systems, computer-aideddesign (CAD) systems and the like. Examples of suitable informationsystems may include transaction processing systems, office systems,decision support systems, knowledge management systems, databasemanagement systems, office information systems. Other examples ofsuitable information systems may include data warehouses, resourceplanning systems, enterprise systems, expert systems, search engines,geographic information systems, global information systems, officeautomation systems and the like.

The resource host system(s) 108 are shown separate from the UNU system100, but in various examples, the UNU system may include one or moreresource host systems. Likewise, while the sensors 106 and displaydevice 110 are shown as part of the UNU system, one or more sensorsand/or display devices may be separate from but in communication withthe UNU system, such as in the case of one or more off-board sensors.

FIG. 2 is an illustration of one example implementation of the UNUsystem 200 in the form of an augmented reality system. As shown, the UNUsystem may include a computer system 202, which in some examples may beconfigured to implement the front-end system 102 and communicationinterface 104 of the UNU system 100 of FIG. 1. In some examples, thecomputer system may comprise, include or be embodied in a portable,hardware-based electronic device such as a smartphone, tablet computerand the like. The computer system may be coupled (by wire or wirelessly)to one or more wearable (e.g., head-mounted) or handheld personaldisplay systems. In some examples, the personal display system may takethe form of augmented reality (AR) eyeglasses, safety goggles, contactlenses and the like (generally AR glasses 204), which may be wearable bya user in an environment. In some examples, the UNU system may beintegrated into the AR glasses.

The computer system 202 and/or AR glasses 204 (or other personal displaysystem) may be equipped with or coupled (by wire or wirelessly) to oneor more sensors (e.g., sensors 106) configured to provide sensed inputto the computer system. As suggested above, for example, the sensors mayinclude an environmental-condition sensor such as a 3D scanner 206configured to scan the user's environment and provide measurements ofpoints on the surfaces of objects in the environment. These measurementsmay be used for a number of different purposes, such as to generate apoint cloud or other 31) model of the user's environment.

The sensors may also include one or more cameras 208 that may beconfigured as a user action-tracking sensor and/orenvironmental-condition sensor. The camera(s) may be configured tocapture images or videos of the user and/or their physical, real-worldenvironment; and in some instances, the camera(s) may thereby capturegestures of the user. In some examples, the sensors may include otheruser action-tracking sensors that in some instances may be configured tocapture gestures of the user, according to any of a number of suitablegesture recognition techniques. Examples of suitable sensors include oneor more accelerometers 210 (motion sensors), gyroscopes 212 (orientationsensors), IMUs 214, fiber-optic sensors 216, muscle-activity (e.g., EMG)sensors 218 and the like. Additionally or alternatively, for example,the sensors may include an audio sensor such as a microphone 220configured to capture audio such as voice commands from the user.

The sensors may take a number of different forms, and in some examplesmay be user wearable such as on the user's finger, wrist, arm, ankle,leg, waist or torso. In some examples, the AR glasses 204 may beequipped with the 3D scanner 206 and camera(s) 208, which may enable the3D scanner and camera(s) to provide measurements and images/video fromthe user's viewpoint. In some examples, the AR glasses and/or computersystem 202 may be equipped with various accelerometer(s) 210,gyroscope(s) 212, IMU(s) 214 and/or microphone 220. In some examples,the computer system may be coupled to a wired glove, armband or otheruser-wearable device or system equipped with various accelerometer(s),gyroscope(s), IMU(s), fiber-optic sensor(s) 216 and/or muscle-activity(e.g., EMG) sensor(s) 218. One example of a suitable wired glove is 5DTData Glove Ultra by Fifth Dimension Technologies. And one example of asuitable armband is the MYO gesture-control armband by Thalmic Labs Inc.

In some examples, the camera 208 and/or microphone 220 may be configuredas physiological-condition sensors configured to provide images/videoand/or audio of a user. Or the computer system 202 and/or AR glasses 204may be equipped with or coupled (by wire or wirelessly) to one or moreother physiological-condition sensors configured to provide sensed inputcharacteristic of the user. This sensed input in various examples maypermit recognition of the user.

In some examples, the computer system 292 and/or AR glasses 204 may beequipped with or coupled (by wire or wirelessly) to one or more positionsensors 222 configured to provide measurements of the position of theuser. As indicated above, examples of suitable position sensors includethose supporting GPS, inertial navigation, Wi-Fi-based positioning,RFID-based positioning or the like.

The AR glasses 294 (or other personal display system) may generate orotherwise enable a live or direct view of the user's environment. The ARglasses may also include a display device 224 (e.g., display device 110)for information from the computer system 202 (e.g., implementing UNUsystem 100), such as information output by electronic resources. In thisregard, the live or direct view of the environment may be augmented byinformation from the computer system. In some examples, the computersystem or AR glasses may further include or be otherwise configured tocommunicate with one or more audio output devices, haptic sensors forproviding audible and/or tactile feedback. This feedback may be providedin addition to or in lieu of certain visual feedback or information bythe display device.

As suggested above and explained below, information from the computersystem 202 may depend on sensed input from one or more sensors 206-220,some of which may be equipped by and others of which may be coupled tothe AR glasses 204 and/or computer system. In one particular example,information from the computer system may vary depending on the viewpointof the user, which may be reflected by sensed input from a sensor suchas the camera 208. In some other examples, the computer system may becoupled to a plurality of AR glasses; and in these examples, informationfrom the computer system may vary for each of the AR glasses dependingon the viewpoints of their respective users.

FIG. 3 illustrates a front-end system 300 according to one exampleimplementation. The front-end system 300 of FIG. 3 may be one example ofthe front-end system 102 of the UNU system 100 of FIG. 1, and which insome examples may be implemented by the computer system 202 of the UNUsystem 200 of FIG. 2. The front-end system may include any of a numberof different subsystems (each an individual system) for performing oneor more functions or operations to carry out user interaction with oneor more electronic resources. As shown, for example, the front-endsystem may include a sensor system 302, analysis system 304, executionsystem 306 and/or evaluation system 308. It should be understood thatwhile the sensor system, analysis system, execution system andevaluation system are shown as part of the front-end system, one or moreof the respective systems may instead be separate from but incommunication with the front-end system or even the UNU system. Further,it should be understood that one or more of the subsystems may functionor operate as a separate system without regard to others of thesubsystems. And it should be understood that the UNU system may includeone or more additional or alternative subsystems than those shown inFIG. 3.

As explained in greater detail below, the sensor system 302 may begenerally configured to receive sensed input from one or more sensors(e.g., sensors 106), which may include information characteristic of auser, the user's environment and/or the user's interaction with theirenvironment. The analysis system 304 may process and correlate thischaracteristic information to existing knowledge to identify an intent,purpose or significance of the characteristic information, and from itssignificance, identify one or more operations that the execution system306 may cause one or more electronic resources to perform. Theevaluation system 308 may perform requisite analysis and return feedbackto improve future operations. Reference will now be made to FIGS. 4, 5,6 and 7, which illustrate more particular examples of a suitable sensorsystem, analysis system, execution system and evaluation system,respectively, according to example implementations of the presentdisclosure.

FIG. 4 illustrates a sensor system 400 according to one exampleimplementation. As indicated above, the sensor system 400 may be oneexample of the sensor system 302 of the front-end system 300 of FIG. 3.The sensor system may be configured to receive and collect sensed inputfrom one or more sensors (e.g., sensors 106). The sensors may begenerally configured to sense a user, the user's environment and/or theuser's interaction with their environment; and their sensed input may becharacteristic of the user, user's environment or user's interactionwith their environment. In some examples, the sensor system may usevarious sensed input to derive information characteristic of the user,user's environment or user's interaction with their environment. Asshown, then, the sensor system may include one or more detectors 402configured to process various sensed input (individually or as acomposite) according to respective detection algorithms to form aderived sensed input. Or in some examples, sensed input may be parsedinto one or more derived sensed input.

The detectors 402 may receive raw sensed input, identify salientinformation and information sets, and provide appropriate computations,transformations and combinations. The UNU system 100 may provide anextensible means to update detectors such as through a bootstrap onstartup, through a need-based dynamic reconfiguration process (based onseveral factors such as device power, potential for tracking items ofinterested and context), and/or through direct operator selection. Insome examples, sensed input from more than one sensor may be used toimplement a detector or to improve the performance of a detector. Anexample of the latter may be the use of an aircraft readiness list (ARL)for a contextually-relevant aircraft, as well as sensed radio frequencyidentification (RFID) and object character recognition (OCR) forline-replaceable unit (LRU) while in an aircraft electrical equipment(EE) bay to detect an object (LRU) and retrieve context-relevantinformation.

As shown in FIG. 4, examples of suitable detectors 402 may include acolor detector, shape detector, object detector, speech detector (e.g.,language/dialect detection, mood detection, stress detection, engagementdetection), face detection (e.g., expression detection, mood detection,stress detection, engagement detection). Examples of suitable detectorsmay also include a motion detector, identity detector, and/or one ormore other detectors such as a feature detector, human characterizer andthe like. In this regard, human characterization may include, forexample, role characterization, geographic and/or temporal localization,language/dialect characterization, stress level (e.g., correlation tomeasurable factors, historical information), engagement detection (e.g.,correlation to measurable factors, historical information), and thelike.

The sensed input including that received (raw sensed input) and anyderived (processed sensed input) by the sensor system 400 may includeinformation characteristic of the user, user's environment or user'sinteraction with their environment. And the sensor system may pass thischaracteristic information to other subsystems of the front-end systemsuch as for analysis, evaluation or the like.

FIG. 5 illustrates an analysis system 500 according to one exampleimplementation, and which may be one example of the analysis system 304of the front-end system 300 of FIG. 3. The analysis system may begenerally configured to receive characteristic information (e.g., rawsensed input, process sensed input) and continuously track and correlatethe respective information to identify its significance (intent), andfrom its significance, operation(s) of electronic resource(s). Forexample, the analysis system may employ object recognition and trackingto identify components on an aircraft, which may then be used to providea dynamically-generated user interface allowing the user to interactwith a component within 3D space (plus time), or identify an item ofinterest in the field of view relative to the user's task.

As shown, the analysis system 500 may include a number of recognizers502 coupled to a persistence store 504 of information models and/orhistorical intent-operation associations. The recognizers may beconfigured to receive and process characteristic information accordingto respective intent algorithms to identify one or more known patterns,which may indicate the significance (intent) of the characteristicinformation. In some examples, one or more recognizers may processcharacteristic information using appropriate information models from thepersistence store. Patterns identified by the analysis system 500 mayinclude those universally-applied (e.g., human-hand object, car object,stop gesture), domain-dependent (e.g. replace 737NG landing gear light,stop special operations force gesture, stop taxing airplane gesture)and/or user-specific (e.g. scheduled break, opportunistic electronicdata transfer, special tools required for disability, user's tasksheet). In some examples, various recognizers 502 may be interdependent.In these examples, interdependencies between recognizers may be contextsensitive such as in instances in which geographical and/or temporalconstraints in conjunction with one or more recognizers influenceresults.

As shown, examples of suitable recognizers include context management,gesture recognition, voice recognition (e.g., text-to-speech), objectrecognition, mood recognition and/or behavior recognition. Otherexamples of suitable recognizers include point-cloud recognition,fatigue recognition, focus recognition, user orientation, toolorientation, object tracking, applied force tracking, task trackingand/or various identifiers (e.g., voice print, fingerprint, heatsignature, facial recognition).

As a more particular example, consider a recognizer 502 for gesturerecognition (gesture recognizer). The gesture recognizer may beconfigured to recognize object (e.g., hand) movement within defined 3Dgeometric zones (e.g., width, length, height) within the user's physicalenvironment. A rectangular cuboid zone may be defined to implement avirtual keyboard on a desk, a square cuboid zone spatially located abovethe rectangular cuboid may be defined to accept other gestures, and asphere spatially located to the right of the rectangular cuboid may bedefined to more directly interface with a building management system tocontrol building lighting. Each zone may be defined in a number ofdifferent manners such as by a width, length and height that may beexpressed in typical coordinates (e.g., x, y, z), but a zone may alsoinclude additional parameters such as date, time, geographic coordinateand the like such as to create situational or context-sensitive userinterface (UI) tailoring.

The persistence store 504 may include storage for one or moreinformation models, past intent-operation pairings, and/or otherinformation according to which characteristic information may beprocessed. The persistence store may provide a historical record ofapproach and success criteria, weighted performance metrics byindividual and/or role and perhaps also categorized by relevant factors(e.g. location, time, availability of equipment, etc.). In one example,information stored by the persistence store may be maintained by anevaluation system, such as the evaluation system 308 of FIG. 3. Combinedwith the recognizers 502, the persistence store may provide a means totrack, gauge and improve performance of the front-end system, andthereby the UNU system.

In a more particular example, characteristic information (e.g., rawsensed input, process sensed input) processed by the recognizers 502 andthe significance identified from the characteristic information may becondensed in a binary form to characteristic-intent pair (CIP). The CIPmay be correlated with operation(s) of electronic resource(s) to form acorrelated operation-CIP pair (OCIP), which in one example maycorrespond to an intent-operation pairing stored by the persistencestore 504. In accordance with example implementations, one or more OCIPsform a unique fingerprint for a sequence of functions ranging fromdetection, to recognition, to operation and so forth. And in someexamples, this fingerprint and/or one or more of its OCIP(s) may beweighted based on one or more performance metrics by individual and/orrole and perhaps also categorized by relevant factors (e.g. location,time, availability of equipment, etc.).

The recognizers 502 may compare the identified significance to thepersistence store 504 for historical significance and associatedoperations based on one or more intent-operation pairings (e.g., one ormore OCIPs that form a unique fingerprint). In one example, eachrecognizer may be configured to compare operations and produce aweighted evaluation based on identification, operation proposal andperhaps other weightings (e.g., user, role). The analysis system 500 maythen pass operational information including the analysis, options,results and/other user feedback (e.g., auditory, visual, touch) to othersubsystems of the front-end system such as for execution of one or moreoperations by one or more electronic resources, and presentation of anycorresponding output from the electronic resource(s).

FIG. 6 illustrates an execution system 600 according to one exampleimplementation, and which may be one example of the execution system 306of the front-end system 300 of FIG. 3. The execution system may begenerally configured to receive operational information and cause one ormore electronic resources to perform one or more operations basedthereon. In various examples, the execution system may cause performanceof operation(s) based on characteristic information (raw sensed input,processed sensed input), prior success patterns, algorithmic operationsand the like. Operations may be dependent upon multiple factors such ascontext, need, urgency, compute capacity, battery life and the like.

The execution system may include a number of operation engines 602 eachof which may be configured to process operational information accordingto a respective algorithm to form an input to one or more electronicresources to cause the respective resource(s) to perform one or moreoperations. In various examples, the operation engines may include thosefor system-derived operations 602 a and/or user-requested operations 602b, although in various instances a system-derived operation may insteadbe user-requested, or a user-requested operation may instead besystem-generated.

The system-derived operation engines 602 a may be generally configuredto cause performance of operations associated with significance (intent)identified from characteristic information, such as by the analysissystem 304. User-requested operation engines 602 b, on the other hand,may be generally configured to cause performance of operationsspecifically requested by the user. As shown, examples of suitablesystem-derived operations include role-based access control (RBAC),tailoring a UI, tailoring information retrieval, tailoring informationdelivery and the like. Examples of system-derived operations may alsoinclude role-based quality assurance monitoring/auditing, audio inputprocessing (e.g., voice command), visual input processing (e.g.,gestures, 3D spatiotemporal processing), movement (or motion) inputprocessing (e.g., gestures), sensor requests and the like. Otherexamples of system-derived operations include automated recording (e.g.,quality control, system record), system calibration, virtualaudio/visual assistant, quality assurance (QA) recording, contexttracking refinement and the like.

Examples of user-requested operations include a request for information,processing, display, communication, documentation and the like. Examplesof user-requested operations may also include a request for UImanipulation, a request to track/monitor an object, a request for anotification/alert, and the like. Other examples of user-requestedoperations include actuator calibration, record audio/video, recordstill images, establish virtual audio/visual connection, record appliedfastener torque, playback recording and the like.

The input formed by an operation engine 602 may be communicated to anappropriate electronic resource hosted by a respective local or remoteresource host system 108 to cause performance of particularoperation(s). The electronic resource may in turn produce an output fromits performance of the operation(s), and return the output to theoperation engine for output by the UNU system 100, such as by itsdisplay device 110, speaker, headphones or the like.

FIG. 7 illustrates an evaluation system 700 according to one exampleimplementation, and which may be one example of the evaluation system308 of the front-end system 300 of FIG. 3. The evaluation system may begenerally configured to evaluate executed operations against successcriteria associated with the operation as well as perhaps its derivationby the analysis system, which may improve performance and efficiency offuture results. As shown, the evaluation system may include a number ofmanagement/evaluation engines 702 coupled to a persistence store 704 ofinformation models and/or historical intent-operation associations. Insome examples, the persistence store 704 may correspond to or otherwisemaintain information stored by the persistence store 504 of the analysissystem 500. Similar to the persistence store 504 of the analysis system,then, the persistence store 704 of the evaluation system may includestorage for past intent-operation pairings, and/or other informationaccording to which characteristic information may be processed. Thepersistence store may provide a historical record of approach andsuccess criteria, weighted performance metrics by individual and/or roleand perhaps also categorized by relevant factors (e.g. location, time,availability of equipment, etc.).

The management/evaluation engines 702 may include management enginessuch as model management and association management engines configuredto manage the information models and intent-operation pairings stored bythe persistence store 704. The management/evaluation engines 702 mayalso include evaluation engines configured to track the significance(intent) of characteristic information, such as that identified by theanalysis system 500. And the management/evaluation engines may includeevaluation engines configured evaluate correlations between thesignificance and operation(s) (e.g., system-derived, user-requested)that electronic resource(s) are caused to perform based thereon, such asby the execution system 600.

As shown, examples of suitable evaluation engines of themanagement/evaluation engines 702 include those for tracking mood,behavior and the like, as well as those for evaluating correlationsbetween mood, behavior and the like and operation(s) of electronicresource(s). Other examples of suitable evaluation engines include thosefor tracking and/or evaluating correlations for context, gestures,voice, objects, point-clouds, fatigue, focus, user orientation, toolorientation, applied force, tasks and the like. In various examples, theevaluations performed by management/evaluation engines may includeproduction of weightings based on the aforementioned performancemetric(s), and which may be applied at various instances to one or moreintent-operation pairings (e.g., OCIPs) stored by the persistence store704 to thereby improve performance and efficiency of future results.

As explained above and in further detail below, the UNU system 100 (andits example UNU system 200) and methods of example implementations mayfacilitate or otherwise provide human-machine interaction in a varietyof different manners. In some examples, one or more sensors 106 (e.g.,sensor(s) 206-222) may be configured to provide sensed input includingmeasurements of motion of a user during performance of a task by theuser. The motion may include a gesture performed in a 3D zone in anenvironment of the user. And the 3D zone may be one of a plurality of 3Dzones in the environment that are defined to accept respective, distinctgestures of the user. In some examples, interpretation of user actionwithin these zones may be further dictated by additional factors, suchas a commanding gesture, current and/or bounding workflow steps and/orpotentially other contextually-relevant actions.

The front-end system 102 (or computer system 202 implementing thefront-end system) may be configured to receive and process the sensedinput including the measurements to identify the gesture and from thegesture, identify operations of an electronic resource (one or moreelectronic resources), which may be hosted by a resource host system 108(one or more resource host systems). In this regard, the front-endsystem may be configured to identify the gesture based on the 3D zone inwhich the gesture is performed. In some more particular examples, theseoperations may be performed by an analysis system 304, 500, or even moreparticularly a gesture recognizer 502 of such an analysis system. Thefront-end system may then be configured to form and communicate an inputto cause the electronic resource to perform the operations and producean output. And the front-end system may be configured to receive theoutput from the electronic resource, and communicate the output to adisplay device 110 (e.g., display device 224), audio output device orhaptic sensor.

The front-end system 102 may be configured to identify the gesture notonly based on the 3D zone, but also based on other factors in which the3D zones or their gestures may differ. For example, the front-end systemmay be configured to identify the gesture further based on the taskperformed by the user. In this example, one or more of the 3D zones orone or more of respective, distinct gestures that the 3D zones aredefined to accept may be different for different tasks. In addition toor in lieu of the task, for example, the front-end system may beconfigured to identify the gesture further based on the userhimself/herself. In this example, one or more of the 3D zones or one ormore of respective, distinct gestures that the 3D zones are defined toaccept may be different for different users.

In some examples, one or more of the 3D zones may be different fordifferent users, with each of the respective 3D zone(s) having adifferent position, shape or size for different users. In at least someof these examples, the UNU system 100 may be configured for use by auser such as through execution of a setup routine. And during thisroutine, the user may provide input from which one or more 3D zones maybe defined, including their positions, shapes and/or sizes, as well asthe gestures that the zones are defined to accept. These and otherparameters unique to the user may then be maintained for use by the UNUsystem, such as in a user profile associated with the respective user.

The UNU system 100 may accept different types of distinct gestures suchas unsupported grand gestures or micro-gestures (first gestures) thatmay depend on the 3D zones, and/or universal gestures (second gestures)that may be accepted without regard to the 3D zones. In variousexamples, universal gestures may be accepted within or across any one ormore of the 3D zones, or even partially or wholly outside of the 3Dzones. In instances in which the user motions a universal gesture, thefront-end system 102 may identify the universal gesture without regardto the 3D zones, whereas again, the front-end system may identify anunsupported grand or micro-gesture based on the 3D zone in which thegesture is performed.

The different types of gestures accepted by the UNU system 100 may bedistinguishable in a number of different manners, including that somedepend on 3D zones and others do not. Unsupported grand gestures may beappropriate for unsupported, outstretched motions and may work betterfor occasional, primarily non-repetitive interfacing, as opposed to moreroutine or frequent operations, for which micro-gestures may workbetter.

Unsupported grand gestures may be appropriate for larger motions havingshorter durations and/or lesser frequencies of use, such as thoselasting no more than 90 seconds, and repeated no more than five times in60 minutes (sometimes referred to as the 90-5 rule). In some examples,unsupported grand gestures may be defined for user control andreconfiguration of workspace—such as moving information from one displaydevice 110 to another, or controlling the position of a rendered objectand/or part. Other unsupported grand gestures may be defined for thepurposes of state transition, such as moving from detection ofcompletion of one operation in a maintenance task and the beginning ofthe next. Other unsupported grand gestures may be defined for the userto provide input from which one or more 3D zones may be defined, asexplained above. For example, one or more unsupported grand gestures maybe defined for the user to specify a point at a particular position fordefining a 3D zone of a particular shape and size having the point atthe particular position as its centroid. And yet other unsupported grandgestures may be defined for interacting with a graphical user interface(GUI), as explained below.

Micro-gestures may be appropriate for smaller motions having longerdurations and/or greater frequencies of use. These gestures may be moreoften defined for motions familiar to a user desiring performance of anexisting operation (e.g., small hand movement to move a digital selectorin a GUI). In some examples, micro-gestures may be more appropriate formotions in which the user has reasonable support for enduringoperations, which may reduce if not minimize fatigue, and increase ifnot maximize utility.

The front-end system 102 (or computer system 202 implementing thefront-end system) may be configured to communicate the output from anelectronic resource to a display device 110 (e.g., display device 224),audio output device or haptic sensor. In some more particular examples,these operations may be performed by an execution system 306, 600, oreven more particularly an operation engine 602 of such an executionsystem. In some examples, the output may be communicated in one of aplurality of distinct desktop environments displayable in respectivefacets of a three-dimensional, multifaceted GUI. In these or othersimilar examples, the output may be communicated to effect rotation ofthe three-dimensional, multifaceted GUI, or to effect selection of oneof desktop environments.

FIGS. 8, 9 and 10 illustrate an example of a 3D, multifaceted GUI 800,and gestures by which a user may interact with the GUI. The GUI mayinclude a plurality of facets 802 in which respective, distinct desktopenvironments may be displayable, and which may be arranged into a 3Dshape. In some examples, the GUI may initially present a single desktopenvironment in a single facet, and then morph into the multifaceted GUIincluding the single desktop environment and other desktop environments,in response to the user performing a gesture defined for this purpose.As shown in FIG. 8, in some of these examples, the gesture may includethe user performing a compression movement with their hands, while theirhands and fingers are extended (and fingertips directed inward), andthumbs vertical.

The multifaceted GUI 800 may be rotatable to enable the user to observevarious views of the desktop environments in its facets 802, such asagain through an appropriate gesture defined for this purpose. As shownin FIG. 9, in one example, this gesture may include the user with theirhands in the aforementioned position shown in FIG. 8, rotating theirhands together about their palms (fingertips from outside arc), withtheir fingertips directed inward. The user may rotate their hands in aparticular direction and with a particular speed to cause the GUI torotate 902 with a related or otherwise corresponding direction andspeed. The user may select one of the desktop environments, which maydirect the GUI to morph back to a single desktop environment in a singlefacet including the selected desktop environment.

In some examples, a desktop environment in a facet 802 of the GUI 800may include a plurality of overlapping windows or displays (generally“windows”), which may be indicated on the facet or GUI. In this regard,a corner of the desktop environment may reveal a series of icons (e.g.,dots) that may be enumerated and keyed to correlate with the number ofwindows of the desktop environment. As shown in FIG. 10, in theseexamples, a topmost window 1002 may be slid 1004 or otherwise moved fromits position to reveal another window 1006 underneath, which again maybe performed in response to the user performing a gesture defined forthis purpose. As shown, one example of a suitable gesture may includethe user with their hands in the aforementioned position shown in FIG.8, transitioning one hand to an outward facing palm and performing aswipe movement with that hand. The underneath window may then become thetopmost window. Or in some examples in which a desktop environment doesnot include overlapping windows, the desktop environment may communicateas much to the user (e.g., flash).

As suggested above, UNU system 100 and methods of exampleimplementations may find use in a variety of potential applications,such as in the context of a complex system such as an aircraft or any ofa number of other structures. A complex system may be generally composedof systems, subsystems, assemblies, subassemblies, components and thelike (each generally a “subsystem”). Each subsystem may be composed ofrespective parts, and each part may include respective features. Theparts of the complex system may be assembled into a number ofsubsystems, which in turn may be assembled into the complex system. Inthe context of an aircraft, one or more parts or subsystems may bedesigned as a modular component of the aircraft often referred to as anLRU, of which a single aircraft may include a number of LRUs and otherparts or subsystems. Any of the complex system itself or any of itssubsystems, parts (of subsystems), features (of parts) or the like mayat times be generally referred to as a “component” or “part” of thecomplex system.

In some examples, the sensors 106 (e.g., sensor(s) 206-222) may beconfigured to provide sensed input including measurements of the motionand/or orientation of the user during performance of the task by theuser to work a complex-system component. In these examples, thefront-end system 102 (or computer system 202 implementing the front-endsystem) may be configured to receive and process the sensed inputincluding the measurements to identify a known pattern that indicates asignificance of the sensed input from which to identify operations ofthe electronic resource. The front-end system may then be configured toform and communicate the input to cause an electronic resource toperform the operations and produce an output, with the operationsincluding determination of an action of the user or calculation of aprocess variable related to performance of the task, from themeasurements. And in these examples, the front-end system may beconfigured to receive the output from the electronic resource, andcommunicate the output to the display device 110 (e.g., display device224), audio output device or haptic sensor.

One more particular example of a suitable application is the support ofengineering activities such as those performed during pre-production,production or post-production of a complex system such as an aircraft.Another example of a suitable application is the support of tasks suchas pre-production, production or post-production tasks performed oncomponents of a complex system. Thus, referring now to FIGS. 11 and 12,example implementations may be used in the context of an aircraftmanufacturing and service method 1100 as shown in FIG. 11, and anaircraft 1200 as shown in FIG. 12. During pre-production, the examplemethod may include specification and design 1102 of the aircraft,manufacturing sequence and processing planning 1104 and materialprocurement 1106. During production, subsystem and componentmanufacturing 1108 and system integration 1110 of the aircraft takesplace. Thereafter, during post-production, the aircraft 1200 may gothrough certification and delivery 1112 in order to be placed in service1114. While in service by a customer (also post-production), theaircraft may be scheduled for routine maintenance and service 1116(which may also include modification, reconfiguration, refurbishment orthe like).

Each of the processes of the example method 1100 may be performed orcarried out by a system integrator, third party and/or operator (e.g.,customer). For the purposes of this description, a system integrator mayinclude for example any number of aircraft manufacturers andmajor-system subcontractors; a third party may include for example anynumber of vendors, subcontractors and suppliers; and an operator mayinclude for example an airline, leasing company, military entity,service organization and the like.

As shown in FIG. 12, an example aircraft 1200 produced by the examplemethod 1100 may include an airframe 1202 with a plurality of systems1204 (each a subsystem or more generally a component) and an interior1206 (including respective components). Examples of high-level systems1204 include one or more of a propulsion system 1208, electrical system1210, hydraulic system 1212, environmental system 1214 and the like. Anynumber of other systems 1204 may be included. Although an aerospaceexample is shown, the principles of the disclosure may be applied toother industries, such as the marine and automotive industries.

The UNU system 100 and method disclosed herein may be specified for useduring any one or more of the stages of the example production andservice method 1100, including any one or more of the pre-production,production and/or post-production stages. Examples are described belowin the context of the implementation of the UNU system shown anddescribed with reference to FIG. 2, namely, in the form of the UNUsystem 200. It should be understood, however, that the scenarios areequally applicable to other more general or more specificimplementations of the UNU system, including those with other types ofpersonal display systems.

During pre-production or production, for example, the UNU system 200 maybe used to enable virtual, on-the-fly designs and engineeringmodifications to mock-ups in real-time, using gesturing in a real-worldenvironment. Its use may thereby simplify, streamline and improve thequality of design work, enhance customer collaboration and buy-in fortheir products, and reduce costs. In one scenario, for example, a user(e.g., engineer) equipped with the UNU system may walk through a mockupof the interior of an aircraft wearing the AR glasses 204, which may beequipped with or coupled to sensor(s) such as a 3D scanner 206, camera208 and the like. The camera may be configured to capture gestures ofthe user. Or in some examples, the UNU system may include one or moreadditional or alternative sensors configured to capture gestures of theuser, such as one or more accelerometers 210 (motion sensors),gyroscopes 212 (orientation sensors), IMUs 214, fiber-optic sensors 216,muscle-activity (e.g., EMG) sensors 218 or the like.

During the walkthrough of the mockup interior, the user may instruct thecomputer system 202 (e.g., through voice commands, gestures, etc.) tocause the 3D scanner 208 to scan and provide measurements of (ormeasure) points on the surfaces of objects in the interior. Moregenerally, the 3D scanner may be caused to provide measurements ofpoints on the surface of an object in the user's environment, orslightly more particularly, provide measurements of points on thesurface of a mockup of a component of the complex system.

The computer system 202 may receive and process sensed input includingthe measurements from the 3D scanner 206 to identify a known patternthat indicates a significance of the sensed input from which to identifyoperations of an appropriate electronic resource. The computer systemmay form and communicate an input to cause the electronic resource toperform the operations. The input may include the measurements, and theoperations may include generation of a point cloud from themeasurements, and transformation of the point cloud to an appropriate 3Dmodel (e.g., 3D CAD model) of the interior (or more generally the objectin the user's environment). More particularly, for example, themeasurements may be input to an appropriate electronic resource that maygenerate a point cloud, and the same or another electronic resource maythen transform the point cloud to an appropriate 3D model of the mockupinterior, which may be stored for future use. In various examples, themockup interior and thus its 3D model may reflect a particular type ofseat configuration such as by wire frames, solids, photo-realistic viewsor the like, and may also reflect other qualities of the mockup interiorsuch as a type of seating, carpet and the like.

Further to the above example scenario, the same or another user equippedwith the UNU system 200 may walk through an empty fuselage of anaircraft wearing the AR glasses 204 including a display device 224. Inanother example, the computer system may be configured to communicatewith a projector in addition to or in lieu of AR glasses' displaydevice. During this walkthrough, the user may instruct the computersystem 202 (e.g., through voice commands, gestures, etc.) to output the3D model of the mockup interior for display in life size dimensions.More generally, the UNU system may be configured to receive an outputincluding the including the 3D model from the electronic resource(s),and communicate the output for display by a display device, where the 3Dmodel may be of a component of a complex system.

The AR glasses 204 may generate or otherwise enable a live or directview of the empty fuselage, which may be augmented by the 3D model ofthe mockup interior from the computer system 202 of the UNU system 200.More generally, the UNU system may be configured to communicate theoutput for display in an instance in which the user's environmentincludes an empty space of the complex system designed for placement ofthe component, with a live or direct view of the empty space beingaugmented by the 3D model. In one example, the UNU system may include orbe coupled to a plurality of AR glasses that include respective displaydevices 224 and are worn by respective users walking through the emptyfuselage (users in the same environment). In this example, the UNUsystem may be instructed to output the 3D model to the display device ofeach of the AR glasses. The 3D model displayed by the respective displaydevices may be anchored to the empty fuselage (users' environment), butvaried for each of the AR glasses depending on viewpoints of therespective users within the fuselage (environment).

The UNU system 200 may further enable a user to make engineering changesto the 3D model of the mockup interior in real-time, which may betterassure user requirements are met. In one example, the user equipped withthe UNU system may use gestures to alter the displayed 3D model, such asby swiping their hand, pinching, zooming or the like to change theconfiguration, shape, attitude, orientation, color combination or thelike of one or more objects of the mockup interior represented by the 3Dmodel. A user may use gestures to pinch a seat and pull outward toexpand it, or swipe to remove it. The gestures may be captured by one ormore appropriate sensors 208-218, and provided (as sensed input) to thecomputer system 202. More generally, an appropriate sensor may beconfigured to capture a gesture of the user indicating alteration to aportion of the 3D model, and the sensed input may include the capturedgesture from the sensor.

The computer system 202 may process the sensed input including thecaptured gesture, and form and communicate an input to cause anappropriate electronic resource to carry out the alteration indicated bythe gesture. The electronic resource may output the altered 3D model oraltered objects (e.g., seats) of the 3D model back to the computersystem. That is, the output may include at least the altered portion ofthe 3D model, which may be in turn output to the display device 224 ofthe AR glasses 204 for display of the 3D model including the alteredportion. The alternations may be shown in real-time as they are made,and may be stored by the electronic resource as an iteration or versionof the 3D model such as for comparison purposes.

In one example, one or more information models reflecting operation ofthe aircraft or one or more of its components may be further drawn in toan appropriate electronic resource, which may simulate operation of theaircraft/components. During or after the simulation, the electronicresource may output results of the simulation to the computer system202, which may in turn output the results to the user through thedisplay device 224 of AR glasses 204. More generally, the UNU system maybe configured to receive an output from an electronic resourcesimulation of operation of the complex system, and communicate theoutput for display by the display device.

In addition to or in lieu of output of the simulation being displayed,the output may be provided to other user interfaces such as audio outputdevice(s), haptic sensor(s) or the like. The audio output device(s) mayprovide audible feedback to the user, and the haptic sensor(s) mayprovide tactile feedback to the user, such as to alert the user ofcertain events during or as a result of the simulation, and may beprovided in addition to or in lieu of visual feedback or otherinformation from the display device 224 of the AR glasses 204. Examplesof suitable events may include collisions or other conflicts orinterference between components, and may facilitate resolution of suchissues.

In some examples, the UNU system 200 may tailor the 3D model and/orother information displayed to a user based on the user's role. That is,the computer system 202 may be configured to communicate the output fordisplay by a plurality of display devices 224 of a respective pluralityof AR glasses 204 worn by respective users, with the output for each ofthe display devices being varied depending on roles of the respectiveusers. In an instance in which a representative of the aircraftmanufacturer and a customer are both wearing AR glasses, the computersystem may cause the representative's AR glasses to display not only the3D model but also availability data, be reminded of customer's pastpreferences or the like, while the customer's AR glasses may displayonly the 3D model. For more information on a compatible system that mayenable viewable information on an as-needed basis, see U.S. patentapplication Ser. No. 13/005,753, entitled: Augmented CollaborationSystem, filed Jan. 13, 2011, the content of which is incorporated byreference in its entirety.

During production or post-production of the aircraft, for example, theUNU system 200 may be used to support a number of activities such asbuild, repair and/or maintenance activities. In various examples, theseand similar activities may be performed according to a work plan thatformalizes instruction for building, creating or otherwise completingrequirements or intention of an engineering design of the aircraft orcomponents of the aircraft. The engineering design may include a set ofrequirements that indicate specifications, tolerances, purposes,materials or other aspects of the aircraft. The work plan may include aseries of work instructions that together comprise a set of directivesto be accomplished by worker(s), and each work instruction may includeone or more tasks. A task may refer to an action or set of actions to beperformed by worker(s) which represents specific and repeatablepurposes, motions or actions of the worker(s).

In some examples, a task may be defined to include or imply one or moredefined actions of the user to work a complex-system component; and forsome tasks, the user may use one or more tools (e.g., hand tools, powertools) to perform the respective actions. For example, the user may betasked with placing, arranging or assembling components, installing oruninstalling components, drilling a hole in a component, installing afastener on a component, cutting through or routing out a portion of acomponent, or the like.

In some examples, a task may have a task definition that includes orimplies one or more defined actions of a type that may subject theworker to ergonomic risk, such as lifting, gripping, working overhead,pushing/pulling, bending, kneeling and the like. Lifting may include theworker lifting an amount of weight some number of times. Gripping mayinclude the worker using a grip (e.g., hand grip, pinch grip,finger/thumb press) to apply an amount of force some number of times.Working overhead may include the worker lifting their hands above thehead some number of times each of which for an amount of time.Pushing/pulling may include the worker pushing/pulling (e.g.,one-handed/two-handed horizontal push/pull, one-armed/two-armed verticalpush/pull) with an amount of force some number of times. Bending mayinclude the worker bending their torso forward (e.g., greater than 45°)some number of times each of which for an amount of time. And similarly,kneeling may include the worker kneeling some number of times each ofwhich for an amount of time.

In some examples, a task definition may specify one or more processrequirements for a user working a complex-system component, which insome instances may involve use of an appropriate tool (e.g., hand tool,power tool). As suggested above, these may include for example,requirements for the placement, arrangement or assembly of component(s),installation or uninstallation of component(s), drilling of a hole in acomponent, installation of a fastener on a component, cutting through orrouting out a portion of a component, or the like. In a more particularexample, a work task for installation of a fastener may includerequirements for process variables related to performance of the tasksuch as torque, swag force and/or preload to be applied to the fastener.

In one scenario, for example, a user (e.g., worker) equipped with theUNU system 200 and wearing the AR glasses 204 may enter a work site tocarry out defined task(s) of a work plan. The computer system 202 and/orAR glasses may be equipped with or coupled to one or more sensors thatmay provide sensed input from which the computer system may recognizethe user. In some examples, the computer system may cause the camera 208to capture an image of the user's face, and/or cause the microphone 220to capture audio of the user, which may be provided to the computersystem as sensed input. Or the computer system may cause otherphysiological-condition sensors to provide sensed input characteristicof the user. The computer system in these examples may be configured toreceive and process the sensed input to recognize the user based onfacial recognition, voice print or the like. The computer system mayfurther communicate with one or more appropriate electronic resourcesbased on the recognized user, such as to clock the user into work,and/or provide an agenda for the day's work including items such as adaily safety message, any reminders of things to watch for that havebeen problems in the past, and the like.

As the user prepares to perform a defined task, the sensors may providesensed input from which the computer system 202 may communicate withappropriate electronic resource(s) to direct the user to where the usermay locate required items for performing the task. The computer systemmay further provide the user with an indication of a known work area forperforming the task, and may output to the display device 224 of the ARglasses 204, a route to the respective area. In one example, thecomputer system may cause the position sensor 222 to provide ameasurement of the position of the user, which may be provided to thecomputer system as sensed input. The computer system may receive andprocess the sensed input to calculate a route from the user's positionto the respective area. As before, in addition to or in lieu of thisdisplay device, the computer system may communicate with a projectorfrom which the output may be displayed (the output in either instanceaugmenting a direct or live view of the user's environment).

The computer system 202 may cause the position sensor 222 to providemeasurements tracking the position of the user as the user moves, whichmay be provided to (as sensed input) and processed by the computersystem to determine when the user approaches the work area. The computersystem 202 may communicate with appropriate electronic resource(s) toprovide reference material or other information relevant to the giventask. As the user arrives at or within proximity of the work area, then,the computer system may output the reference material and other relevantinformation to the display device 224 of the AR glasses 204, such as ina sequential, context-sensitive manner.

While the user is at or within proximity of the work area, computersystem 202 may cause the camera 208 to automatically capture images orvideo of the user and/or their environment, such as from the user'sviewpoint. As the user's viewpoint turns to an area of an object withwhich the user must interact to perform the task, the computer systemmay receive and process sensed input to recognize the object,communicate with appropriate electronic resource(s) and output visualcues that augment the user's direct view to highlight the respectiveobject. In some examples, the visual cue may have characteristics (e.g.,color, shading, pattern, intensity, direction) that may be customized toreflect a status of an object. The computer system may further outputstep-by-step tasks/actions (visually and/or audibly), and may at variousinstances output further visual assistance. This visual/audibleassistance may include, for example, assistive videos illustratingperformance of a task/action, or an animated view of componentsassembling, perhaps with voice overs that describe the tasks/actionsbeing performed, assistive notes, a rationale behind the tasks/actionsor the like.

The user may instruct the computer system 202 (e.g., through voicecommands, gestures, etc.) to communicate with appropriate electronicresource(s) to provide information regarding an object of interest,which the computer system may then output to the display device 224 ofthe AR glasses 204. For example, the user could speak a voice commandsuch as “find flap slat electronics unit,” which may be captured by themicrophone 220 and processed to cause the computer system to communicatewith appropriate electronic resource(s) and output a visual cue thataugment the user's direct view to highlight the respective flap slatelectronics unit (an LRU). Automatically or in response to userinstruction, the computer system may be caused to customizecharacteristics of the visual cue to reflect the status of the LRU, orprovide information regarding the LRU.

As the user performs the task, the computer system 202 may cause varioussensors to provide sensed input from which the computer system may trackperformance of the task. For example, the computer system may cause thecamera 208 to automatically capture video to as the user performs thetask. Additionally or alternatively, for example, the computer systemmay cause the accelerometer 210, gyroscope 212, IMU 214, fiber-opticsensor 216, muscle-activity (e.g., EMG) sensor 218 or other appropriatesensor to provide measurements of motion and/or orientation of the user,which may be characteristic of one or more actions of the user. This mayinclude measurements characteristic of the user lifting, gripping,working overhead, pushing/pulling, bending, kneeling and the like. Insome examples, the muscle-activity sensor may provide measurements ofthe user's muscle activity, from which a lifted weight, grip forceand/or push/pull force may be calculated.

As or after the computer system 202 tracks performance of a task, thecomputer system may output the video and/or other sensed input to anappropriate electronic resource for storage and recall, such as for anumber of different purposes. For example, the computer system mayoutput the video and/or other sensed input to an electronic resource forperformance evaluation purposes. As indicated above, a task may bedefined to include or imply defined actions of the user. As alsoindicated above, the sensed input may include measurements of motionand/or orientation of the user, which may be characteristic of one ormore actions of the user. In some examples, then, computer system mayoutput the video and/or other sensed input from which the electronicresource may determine one or more actions of the user duringperformance of the task, and compare the determined actions torespective defined actions of the task definition. The electronicresource may determine compliance of the user's performance of the taskwith its definition based on the comparison, which may thereby enableevaluation of the user's performance.

In some examples, the computer system 202 may output the video and/orother sensed input to an electronic resource for ergonomic evaluationpurposes. As suggested above, motion and/or orientation of the user maybe characteristic of different types of action some of which may subjectthe worker to ergonomic risk. The computer system may therefore providethe video and/or other sensed input to the electronic resource, whichmay be configured to perform or facilitate performance of an ergonomicrisk assessment based on the video and/or other sensed input. In oneexample, the ergonomic risk assessment may be performed in a mannersimilar to that described by U.S. Pat. No. 7,457,678, entitled: Methodfor Managing Ergonomic Risk Exposure in Manufacturing, issued Nov. 25,2008, the content of which is incorporated by reference in its entirety.

In some examples, the computer system 202 may output the video and/orother sensed input to an electronic resource for inspection purposes. Assuggested above, a task definition may specify process requirement(s)for the user working a complex-system component. For example, a task forinstallation of a fastener may include requirements for processvariables such as torque, swag force and/or preload to be applied to thefastener. In some examples, then, computer system may output themeasurements of muscle activity (in addition to or in lieu of videoand/or other sensed input) from which the electronic resource maycalculate a number of different process variables from themeasurements—including torque, swag force and/or preload applied to thefastener by the user (directly or using a tool). The electronic resourcemay compare the calculated process variables to respective specifiedprocess requirements of the task definition. The electronic resource maydetermine compliance of the calculated process variables with respectivespecified process requirements based on the comparison, which maythereby enable inspection of task against its process requirements.

An electronic resource may produce and output results of its evaluationor inspection as or after the user performs the task. In instances inwhich the results are output as the user performs the task, the resultsmay facilitate the user's proper performance of the task, such asaccording to its definition and specified process requirements. Ineither instance, however, the results may facilitate the same or anotheruser's proper performance of the same or another task that may depend onthe respective task. In some examples, automated recognition of taskcompletion may drive pacer boards, progress dashboards, completionmetric software and the like, which may help improve real-timeproduction status.

The electronic resource may produce any of a number of different resultsof an evaluation or inspection. For example, the electronic resource mayproduce one or more indications of the determined action of the user,the respective action of the task definition, and/or compliance of theuser's performance of the task with its definition based on theircomparison. In another example, the electronic resource may produce oneor more indications of the calculated process variables (e.g., torque,swag force, preload), respective process requirements specified by thetask definition, and/or compliance of the calculated process variableswith the respective process requirements based on their comparison. Insome examples, the indication of compliance with a task definition orprocess requirements may be reflected in multiple relative levels thatmay be distinguished by respective ranges of compliance, such as fullycompliant, partially compliant or non-compliant.

In some examples, the electronic resource may output results of theevaluation or inspection to the computer system 202, which may in turnoutput the results to the user through the display device 224 of ARglasses 204, and/or through other user interfaces (e.g., audio outputdevice, haptic sensor). The results may be output as a numerical ortextual feedback. Additionally or alternatively, the results may beoutput as visual, audible and/or haptic cues that in some instances mayreflect a level of compliance to a task definition or processrequirement. In the context of fastener installation, for example, thecomputer system may output a visual cue, audible feedback and/or hapticfeedback to the user, such as to alert the user that the calculatedprocess variable (e.g., torque, swag force, preload) is compliant withthe specified process requirement.

Evaluation, inspection and other similar operation(s) performed in amanner of example implementations may also enable the user to keepfocused on their task instead of waiting for an evaluation, inspectionor other process, which may result in a significant time savings, betterbuild quality and/or reduction of cognitive disruption to the user.Moreover, in the event the user takes a break or is otherwiseinterrupted (e.g., the UNU system 200 experiences a failure), the user'senvironment may be captured. On returning to the task, the video may berecalled to assist the user returning to the point at which they wereinterrupted, procedurally with context intact.

Storage of records of tasks including video and other information mayenable a number of other processes in addition to or in lieu of thosedescribed above. An electronic resource using data analytics may, forexample, scan these records looking for aberrations, as well asimprovement that may be automatically included in futureprocess-improvement activities. The time each instance of a task isperformed may be recorded and output for display in a composite, living“value stream map” that may highlight new areas of opportunity forimprovement in various tasks.

The UNU system 200 may further enable the user on-the-fly access toother users (e.g., engineers, planners) via appropriate electronicresources. The user may thereby collaborate with or otherwise receiveassistance from others to address a task or problem with the task.During this collaboration, the user may instruct the computer system 202to provide the other users with contextual or other supportinginformation that may facilitate their interaction. Even during inservice of the aircraft, this may enable collaboration among staff. Forexample, a flight attendant equipped with the UNU system could speak avoice command such as “I need more coffee” to cause the computer systemto communicate with appropriate electronic resource(s) to locate anotherflight attendant or airline staff nearest the coffee, and output analert or other indication of the one attendant's need for coffee. Theaugmented reality system in this scenario may also benefit from aspectsof the system disclosed in the aforementioned '753 application.

Moreover, as suggested above, the UNU system 200 may direct a user towhere they may locate required items for performing a task. This mayinclude inventoried items, and the computer system 202 may communicatewith appropriate electronic resources to perform a variety of activitiesfor inventoried items. Examples of suitable activities include inventorycontrol and auto-reconcile consumption activities, set/reset min/maxinventory levels, reorder points and the like, pay for inventory onconsumption, and the like. This may synchronize the supply chain to apull signal of consumption, which may facilitate just-in-time supply.For more information on various min/max aspects, see U.S. Pat. No.7,769,643, entitled: Min/Max Inventory Control System and AssociatedMethod and Computer Program Product, issued Aug. 3, 3010; and U.S.Patent Application Publication No. 2011/0029412, entitled: Apparatus andMethod for Controlling Inventory, published Feb. 3, 2011, the contentsof both of which are hereby incorporated by reference.

The UNU system 100 (and its example UNU system 200) and method ofexample implementations may also find use in a variety of otherpotential applications. The UNU system may have application ininformation kitting in which the virtual and physical may be combined,such as by augmenting the user's physical environment with virtualinformation elements at the time and point of use, and in acontext-sensitive manner.

In another example, the UNU system 100 of example implementations mayhave application in accident investigations. In this regard, a pointcloud may be recorded and generated from a debris field for analysisthat may model differing algorithms to inform an investigation as towhich of a myriad of potential events is at the root of the situation athand.

In yet another example, the UNU system 100 may have application inemotion, user intent/behavior. Gestures, verbal emotional cues, microexpressions and the like may be elements for context. This informationmay be folded back into contextual information to inform predictivecapabilities for need. Engagement may be judged, information and/or auser interface may be updated (provide more help/information), and/or atie may be made to an available expert (recall what worked last time).

In other examples, the UNU system 100 may have application in virtualguided assistance; and/or mobile, mesh networks, for borrowing computingresources. And in yet another example, the UNU system may haveapplication in human body route tracking, such as to generate an alertif designated work zone has not been entered, or if a dwell time has notmet anticipated threshold for a particular operation.

According to example implementations of the present disclosure, the UNUsystem 100 and resource host systems 108, and their respectivesubsystems may be implemented by various means. Similarly, the UNUsystem 200, and examples of a front-end system 300, sensor system 400,analysis system 500, execution system 600 and evaluation system 700,including each of their respective subsystems, elements and the like(e.g., computer system 202), may be implemented by various meansaccording to example implementations. Means for implementing thesystems, subsystems and their respective elements may include computerhardware, alone or under direction of one or more computer program codeinstructions, program instructions or executable computer-readableprogram code instructions from a computer-readable storage medium.

In one example, one or more apparatuses may be provided that areconfigured to function as or otherwise implement the systems, subsystemsand respective elements shown and described herein. In examplesinvolving more than one apparatus, the respective apparatuses may beconnected to or otherwise in communication with one another in a numberof different manners, such as directly or indirectly via a wired orwireless network and the like.

Generally, an apparatus of exemplary implementations of the presentdisclosure may comprise, include or be embodied in one or more fixed orportable, hardware-based electronic devices. Examples of suitableelectronic devices include a smartphone, tablet computer, laptopcomputer, desktop computer, workstation computer, server computer andthe like. The apparatus may include one or more of each of a number ofcomponents such as, for example, a processor connected to a memory(e.g., storage device).

The processor is generally any piece of computer hardware that iscapable of processing information such as, for example, data,computer-readable program code, instructions and the like (generally“computer programs,” e.g., software, firmware, etc.), and/or othersuitable electronic information. More particularly, for example, theprocessor may be configured to execute computer programs, which may bestored onboard the processor or otherwise stored in the memory (of thesame or another apparatus). The processor may be a number of processors,a multi-processor core or some other type of processor, depending on theparticular implementation. Further, the processor may be implementedusing a number of heterogeneous processor systems in which a mainprocessor is present with one or more secondary processors on a singlechip. As another illustrative example, the processor may be a symmetricmulti-processor system containing multiple processors of the same type.In yet another example, the processor may be embodied as or otherwiseinclude one or more application-specific integrated circuits (ASICs),field-programmable gate arrays (FPGAs) and the like. Thus, although theprocessor may be capable of executing a computer program to perform oneor more functions, the processor of various examples may be capable ofperforming one or more functions without the aid of a computer program.

The memory is generally any piece of computer hardware that is capableof storing information such as, for example, data, computer programsand/or other suitable information either on a temporary basis and/or apermanent basis. The memory may include volatile and/or non-volatilememory, and may be fixed or removable. Examples of suitable memoryinclude random access memory (RAM), read-only memory (ROM), a harddrive, a flash memory, a thumb drive, a removable computer diskette, anoptical disk, a magnetic tape or some combination of the above. Opticaldisks may include compact disk-read only memory (CD-ROM), compactdisk-read/write (CD-R/W), DVD and the like. In various instances, thememory may be referred to as a computer-readable storage medium which,as a non-transitory device capable of storing information, may bedistinguishable from computer-readable transmission media such aselectronic transitory signals capable of carrying information from onelocation to another. Computer-readable medium as described herein maygenerally refer to a computer-readable storage medium orcomputer-readable transmission medium.

In addition to the memory, the processor may also be connected to one ormore interfaces for transmitting, receiving and/or outputtinginformation. The interfaces may include one or more communicationsinterfaces 104, sensors 106 (e.g., sensors 206-220) and/or userinterfaces, examples of which are described above with reference tocommunications interface 104, sensor 106 and/or user interface(including display device 108—e.g., display device 224).

As indicated above, program code instructions may be stored in memoryand executed by a processor to implement functions of the systems,subsystems and their respective elements described herein. As will beappreciated, any suitable program code instructions may be loaded onto acomputer or other programmable apparatus from a computer-readablestorage medium to produce a particular machine, such that the particularmachine becomes a means for implementing the functions specified herein.These program code instructions may also be stored in acomputer-readable storage medium that can direct a computer, a processoror other programmable apparatus to function in a particular manner tothereby generate a particular machine or particular article ofmanufacture. The instructions stored in the computer-readable storagemedium may produce an article of manufacture, where the article ofmanufacture becomes a means for implementing functions described herein.The program code instructions may be retrieved from a computer-readablestorage medium and loaded into a computer, processor or otherprogrammable apparatus to configure the computer, processor or otherprogrammable apparatus to execute operations to be performed on or bythe computer, processor or other programmable apparatus.

Retrieval, loading and execution of the program code instructions may beperformed sequentially such that one instruction is retrieved, loadedand executed at a time. In some example implementations, retrieval,loading and/or execution may be performed in parallel such that multipleinstructions are retrieved, loaded, and/or executed together. Executionof the program code instructions may produce a computer-implementedprocess such that the instructions executed by the computer, processoror other programmable apparatus provide operations for implementingfunctions described herein.

Execution of instructions by a processor, or storage of instructions ina computer-readable storage medium, supports combinations of operationsfor performing the specified functions. It will also be understood thatone or more functions, and combinations of functions, may be implementedby special purpose hardware-based computer systems and/or processorswhich perform the specified functions, or combinations of specialpurpose hardware and program code instructions.

Many modifications and other implementations of the disclosure set forthherein will come to mind to one skilled in the art to which thisdisclosure pertains having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the disclosure is not to be limited to the specificimplementations disclosed and that modifications and otherimplementations are intended to be included within the scope of theappended claims. Moreover, although the foregoing descriptions and theassociated drawings describe example implementations in the context ofcertain example combinations of elements and/or functions, it should beappreciated that different combinations of elements and/or functions maybe provided by alternative implementations without departing from thescope of the appended claims. In this regard, for example, differentcombinations of elements and/or functions than those explicitlydescribed above are also contemplated as may be set forth in some of theappended claims. Although specific terms are employed herein, they areused in a generic and descriptive sense only and not for purposes oflimitation.

What is claimed is:
 1. A ubiquitous natural user system comprising: oneor more sensors configured to provide sensed input includingmeasurements of motion of a user during performance of a task by theuser, the motion including a gesture performed in a three-dimensional(3D) zone in an environment of the user, the 3D zone being one of apluralitv of 3D zones in the environment that are defined to acceptrespective, distinct gestures of the user; and a front-end systemcoupled to the one or more sensors, and configured to receive andprocess the sensed input including the measurements to identify thegesture and from the gesture, identify operations of an electronicresource, the front-end system being configured to identify the gesturebased on the one of the plurality of 3D zones in which the gesture isperformed, wherein the front-end system is configured to form andcommunicate an input to cause the electronic resource to perform theoperations and produce an output, wherein the front-end system isconfigured to receive the output from the electronic resource, andcommunicate the output to a display device, audio output device orhaptic sensor, wherein the gesture is a first gesture, and the motionfurther includes a second gesture performed in the environment of theuser and that is distinct from the respective, distinct gestures thatthe plurality of 3D zones are defined to accept, and wherein thefront-end system being configured to receive and process the sensedinput includes being configured to receive and process the sensed inputto further identify the second gesture, the operations of the electronicresource including operations identified from the first gesture andoperations identified from the second gesture, the front-end systembeing configured to identify the second gesture without regard to theplurality of 3D zones.
 2. The ubiquitous natural user system of claim 1,wherein the front-end system being configured to receive and process thesensed input includes being configured to receive and process the sensedinput including the measurements to identify the gesture further basedon the task, one or more of the plurality of 3D zones or one or more ofrespective, distinct gestures that the plurality of 3D zones are definedto accept being different for different tasks.
 3. The ubiquitous naturaluser system of claim 1, wherein the front-end system being configured toreceive and process the sensed input includes being configured toreceive and process the sensed input including the measurements toidentify the gesture further based on the user, one or more of theplurality of 3D zones or one or more of respective, distinct gesturesthat the plurality of 3D zones are defined to accept being different fordifferent users.
 4. The ubiquitous natural user system of claim 3,wherein one or more of the plurality of 3D zones are different fordifferent users, each of the one or more of the plurality of 3D zoneshaving a different position, shape or size for different users.
 5. Amethod comprising: providing by one or more sensors, sensed inputincluding measurements of motion of a user during performance of a taskby the user, the motion including a gesture performed in athree-dimensional (3D) zone in an environment of the user, the 3D zonebeing one of a puralitv of 3D zones in the environment that are definedto accept respective, distinct gestures of the user; receiving andprocessing the sensed input including the measurements to identify thegesture and from the gesture, identify operations of an electronicresource, the gesture being identified based on the one of the pluralityof 3D zones in which the gesture is performed; forming and communicatingan input to cause the electronic resource to perform the operations andproduce an output; and receiving the output from the electronicresource, and communicating the output to a display device, audio outputdevice or haptic sensor, wherein the gesture is a first gesture, and themotion further includes a second gesture performed in the environment ofthe user and that is distinct from the respective, distinct gesturesthat the plurality of 3D zones are defined to accept, and whereinreceiving and processing the sensed input includes receiving andprocessing the sensed input to further identify the second gesture, theoperations of the electronic resource including operations identifiedfrom the first gesture and operations identified from the secondgesture, the second gesture being identified without regard to theplurality of 3D zones.
 6. The method of claim 5, wherein receiving andprocessing the sensed input includes receiving and processing the sensedinput including the measurements to identify the gesture further basedon the task, one or more of the plurality of 3D zones or one or more ofrespective, distinct gestures that the plurality of 3D zones are definedto accept being different for different tasks.
 7. The method of claim 5,wherein receiving and processing the sensed input includes receiving theprocessing the sensed input including the measurements to identify thegesture further based on the user, one or more of the plurality of 3Dzones or one or more of respective, distinct gestures that the pluralityof 3D zones are defined to accept being different for different users.8. The method of claim 7, wherein one or more of the plurality of 3Dzones are different for different users, each of the one or more of theplurality of 3D zones having a different position, shape or size fordifferent users.
 9. A computer-readable storage medium that isnon-transitory and has computer-readable program code stored thereinthat, in response to execution by a processor, cause an apparatus to atleast: receive from one or more sensors, sensed input includingmeasurements of motion of a user during performance of a task by theuser, the motion including a gesture performed in a three-dimensional(3D) zone in an environment of the user, the 3D zone being one of aplurality of 3D zones in the environment that are defined to acceptrespective, distinct gestures of the user; receive and process thesensed input including the measurements to identify the gesture and fromthe gesture, identify operations of an electronic resource, the gesturebeing identified based on the one of the plurality of 3D zones in whichthe gesture is performed; form and communicate an input to cause theelectronic resource to perform the operations and produce an output; andreceive the output from the electronic resource, and communicate theoutput to a display device, audio output device or haptic sensor,wherein the gesture is a first gesture, and the motion further includesa second gesture performed in the environment of the user and that isdistinct from the respective, distinct gestures that the plurality of 3Dzones are defined to accept, and wherein the apparatus being caused toreceive and process the sensed input includes being caused to receiveand process the sensed input to further identify the second gesture, theoperations of the electronic resource including operations identifiedfrom the first gesture and operations identified from the secondgesture, the apparatus being caused to identify the second gesturewithout regard to the plurality of 3D zones.
 10. The computer-readablestorage medium of claim 9, wherein the apparatus being caused to receiveand process the sensed input includes being caused to receive andprocess the sensed input including the measurements to identify thegesture further based on the task, one or more of the plurality of 3Dzones or one or more of respective, distinct gestures that the pluralityof 3D zones are defined to accept being different for different tasks.11. The computer-readable storage medium of claim 9, wherein theapparatus being caused to receive and process the sensed input includesbeing caused to receive and process the sensed input including themeasurements to identify the gesture further based on the user, one ormore of the plurality of 3D zones or one or more of respective, distinctgestures that the plurality of 3D zones are defined to accept beingdifferent for different users.
 12. The computer-readable storage mediumof claim 11, wherein one or more of the plurality of 3D zones aredifferent for different users, each of the one or more of the pluralityof 3D zones having a different position, shape or size for differentusers.
 13. A computer-readable storage medium that is non-transitory andhas computer-readable program code stored therein that, in response toexecution by a processor, cause an apparatus to at least: receive fromone or more sensors, sensed input including measurements of motion of auser during performance of a task by the user, the motion including agesture performed in a three-dimensional (3D) zone in an environment ofthe user, the 3D zone being one of a plurality of 3D zones in theenvironment that are defined to accept respective, distinct gestures ofthe user; receive and process the sensed input including themeasurements to identify the gesture and from the gesture, identifyoperations of an electronic resource, the gesture being identified basedon the one of the plurality of 3D zones in which the gesture isperformed; form and communicate an input to cause the electronicresource to perform the operations and produce an output; and receivethe output from the electronic resource, and communicate the output to adisplay device, audio output device or haptic sensor, wherein theapparatus being caused to communicate the output includes being causedto communicate the output to the display device, the output beingcommunicated in one of a plurality of distinct desktop environmentsdisplayable in respective facets of a three-dimensional, multifacetedgraphical user interface.
 14. A computer-readable storage medium that isnon-transitory and has computer-readable program code stored thereinthat, in response to execution by a processor, cause an apparatus to atleast: receive from one or more sensors, sensed input includingmeasurements of motion of a user during performance of a task by theuser, the motion including a gesture performed in a three-dimensional(3D) zone in an environment of the user, the 3D zone being one of aplurality of 3D zones in the environment that are defined to acceptrespective, distinct gestures of the user; receive and process thesensed input including the measurements to identify the gesture and fromthe gesture, identify operations of an electronic resource, the gesturebeing identified based on the one of the plurality of 3D zones in whichthe gesture is performed; form and communicate an input to cause theelectronic resource to perform the operations and produce an output; andreceive the output from the electronic resource, and communicate theoutput to a display device, audio output device or haptic sensor,wherein the apparatus being caused to communicate the output includesbeing caused to communicate the output to the display device, the outputbeing communicated to effect rotation of a three-dimensional,multifaceted graphical user interface having a plurality of distinctdesktop environments displayable in respective facets thereof, or toeffect selection of one of the plurality of distinct desktopenvironments.
 15. A ubiquitous natural user system comprising: one ormore sensors configured to provide sensed input including measurementsof motion of a user during performance of a task by the user, the motionincluding a gesture performed in three-dimensional (3D) zone in anenvironment of the user, the 3D zone being one of a plurality of 3Dzones in the environment that are defined to accept respective, distinctgestures of the user; and a front-end system coupled to the one or moresensors, and configured to receive and process the sensed inputincluding the measurements to identify the gesture and from the gesture,identify operations of an electronic resource, the front-end systembeing configured to identify the gesture based on the one of theplurality of 3D zones in which the gesture is performed, wherein thefront-end system is configured to form and communicate an input to causethe electronic resource to perform the operations and produce an output,wherein the front-end system is configured to receive the output fromthe electronic resource, and communicate the output to a display device,audio output device or haptic sensor, and wherein the front-end systembeing configured to communicate the output includes being configured tocommunicate the output to the display device, the output beingcommunicated in one of a plurality of distinct desktop environmentsdisplayable in respective facets of a three-dimensional, multifacetedgraphical user interface.
 16. A ubiquitous natural user systemcomprising: one or more sensors configured to provide sensed inputincluding measurements of motion of a user during performance of a taskby the user, the motion including a gesture performed in athree-dimensional (3D) zone in an environment of the user, the 3D zonebeing one of a plurality of 3D zones in the environment that are definedto accept respective, distinct gestures of the user; and a front-endsystem coupled to the one or more sensors, and configured to receive andprocess the sensed input including the measurements to identify thegesture and from the gesture, identify operations of an electronicresource, the front-end system being configured to identify the gesturebased on the one of the plurality of 3D zones in which the gesture isperformed, wherein the front-end system is configured to form andcommunicate an input to cause the electronic resource to perform theoperations and produce an output, wherein the front-end system isconfigured to receive the output from the electronic resource, andcommunicate the output to a display device, audio output device orhaptic sensor, and wherein the front-end system being configured tocommunicate the output includes being configured to communicate theoutput to the display device, the output being communicated to effectrotation of a three-dimensional, multifaceted graphical user interfacehaving a plurality of distinct desktop environments displayable inrespective facets thereof, or to effect selection of one of theplurality of distinct desktop environments.
 17. A method comprising:providing by one or more sensors, sensed input including measurements ofmotion of a user during performance of a task by the user, the motionincluding a gesture performed in a three-dimensional (3D) zone in anenvironment of the user, the 3D zone being one of a plurality of 3Dzones in the environment that are defined to accept respective, distinctgestures of the user; receiving and processing the sensed inputincluding the measurements to identify the gesture and from the gesture,identify operations of an electronic resource, the gesture beingidentified based on the one of the plurality of 3D zones in which thegesture is performed; forming and communicating an input to cause theelectronic resource to perform the operations and produce an output; andreceiving the output from the electronic resource, and communicating theoutput to a display device, audio output device or haptic sensor,wherein communicating the output includes communicating the output tothe display device, the output being communicated in one of a pluralityof distinct desktop environments displayable in respective facets of athree-dimensional, multifaceted graphical user interface.
 18. A methodcomprising: providing by one or more sensors, sensed input includingmeasurements of motion of a user during performance of a task by theuser, the motion including a gesture performed in a three-dimensional(3D) zone in an environment of the user, the 3D zone being one of aplurality of 3D zones in the environment that are defined to acceptrespective, distinct gestures of the user; receiving and processing thesensed input including the measurements to identify the gesture and fromthe gesture, identify operations of an electronic resource, the gesturebeing identified based on the one of the plurality of 3D zones in whichthe gesture is performed; forming and communicating an input to causethe electronic resource to perform the operations and produce an output;and receiving the output from the electronic resource, and communicatingthe output to a display device, audio output device or haptic sensor,wherein communicating the output includes communicating the output tothe display device, the output being communicated to effect rotation ofa three-dimensional, multifaceted graphical user interface having aplurality of distinct desktop environments displayable in respectivefacets thereof, or to effect selection of one of the plurality ofdistinct desktop environments.